There must be someone without all that baggage. He has been dishonest (he has to be or he loses his family). He has to choose between you and all the rest of his life. He will be conflicted and unhappy if he does not make that decision.
At best you will only have half a man until he makes the break.
He makes the break or you are in limbo.